Statistical Phrases in Automated Text Categorization

In this work we investigate the usefulness of n-grams for document indexing in text categorization (TC). We call n-gram a set tk of n word stems, and we say that tk occurs in a document dj when a sequence of words appears in dj that, after stop word removal and stemming, consists exactly of the n stems in tk, in some order. Previous researches have investigated the use of n-grams (or some variant of them) in the context of specific learning algorithms, and thus have not obtained general answers on their usefulness for TC. In this work we investigate the usefulness of n-grams in TC independently of any specific learning algorithm. We do so by applying feature selection to the pool of all α-grams (α ≤ n), and checking how many n-grams score high enough to be selected in the top σ α-grams. We report the results of our experiments, using several feature selection functions and varying values of σ, performed on the Reuters-21578 standard TC benchmark. We also report results of making actual use of the selected n-grams in the context of a linear classifier induced by means of the Rocchio method. Categories and subject descriptors: H.3.3 [Information storage and retrieval]: Information search and retrieval Information filtering; H.3.3 [Information storage and retrieval]: Systems and software Performance evaluation (efficiency and effectiveness); I.2.3 [Artificial Intelligence]: Learning Induction Terms: Algorithms, Experimentation, Theory